Evaluating embedding based retrieval beyond historical search results

Embedding based retrieval (EBR; a.k.a. vector search) provides an efficient implementation of semantic search and has seen wide adoption in e-commerce. While EBR models are often trained on historical user-engagement data that signifies query-item relevance, to select A/B test candidates, we need independent metrics to predict the relevance of their recall that expands null and low search results in production. This is because with null and low queries, an item’s low engagement history might reflect not its low relevance, but the failure of the existing search engine.

This talk presents a number of ways to leverage organizational knowledge to evaluate the quality of query and item embeddings, and how well EBR on top of those may improve on relevance while expanding the recall.

Yu Cao

eBay

Yu Cao is an Applied Researcher at eBay. His work involves developing and deploying ML solutions to real-world problems faced in e-commerce search. Before joining eBay Yu was a PhD in linguistics, where his research took a primary interest in uniting formal theories with learning models.